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Juliana  S.  Lancaster 
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Georgia  Institute  of  Technology 
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Abstract 

Problem  solving  Is  known  to  vary  In  some  predictable  ways  as  a  function  of  experience.  In  this  study,  we  have 
Investigated  the  effects  of  experience  on  the  problem  solving  behavior  and  knowledge  base  of  workers  In  an  applied 
setting:  automobile  mechanics.  The  automobile  Itself  la  a  highly  complex  system  with  many  Interconnected  subsys¬ 
tems.  Problem  descriptions  (l.e.,  symptoms)  presented  to  a  mechanic  who  needs  to  diagnose  a  car,  however,  are 
usually  quits  sketchy,  requiring  the  collection  of  mors  Information  betas  solution.  Novices  are  leas  able  than  experts 
to  diagnoae  any  but  the  obvious  problems,  and  we  are  Interested  In  Identifying  the  qualitative  dMarencea  between 
mechanics  at  different  levels  of  expertise.  In  the  study  r  spa  ted,  we  obesrved  three  student  mechanics  In  a  post- 
secondary  technical  school,  each  at  a  different  level  of  expertise,  diagnose  six  problems  Introduced  Into  cers  In  the 
school.  We  then  analyzed  the  protocols  we  ooiiected  to  find  the  knowledge  and  ska  teg  let  used  In  solving  each  prob¬ 
lem.  We  also  analyzed  the  series  of  protocols  fa  each  student  to  find  the  changes  In  knowledge  and  strategies  used 
In  solving  lata  problems  as  compared  to  earlier  problems.  Differences  wae  seen  In  both  the  knowledge  used  by  the 
subjects  and  in  their  general  approach  to  diagnosis.  As  a  result  of  experience,  the  student  mechanics  seemed  to 
Improve  In  three  areas:  (1)  their  knowledge  of  the  relationships  between  symptoms  and  possible  failures  was  aug¬ 
mented,  (2)  their  causal  models  of  the  car's  systems  was  augmented,  and  (3)  their  general  kouble shooting  pro¬ 
cedures  and  decision  rules  was  much  Improved. 


1.  Introduction 

Problem  solving  Is  known  to  very  In  some  predictable  ways  as  a  function  of  expertise.  When  the  process  of 
problem  solving  first  came  unda  scrutiny  by  psychology  and  computer  science  researchers,  the  problems  studied 
wae  in  knowledge-lean  domains  In  which  well-defined  situations  have  known  solutions  (Reed,  Ernst,  A  Banal  I,  1074; 
Reed  &  Johnson,  1977;  Reltman,  1976;  Simon.  1975).  In  that  work,  the  behavior  of  Interest  was  genaalfy  a  variable 
such  aa  numba  of  steps  to  completion  a  numba  of  correct  solutions.  Recently  however,  Inlaest  In  problem  solving 
has  leaned  more  toward  problems  In  knowledge-rich  domains  such  as  physics  (Chi,  Glasa.A  Rees,  1982;  Simon  & 
Simon,  1978),  thamodynamics  (Bhaskar  A  Simon,  1977),  architecture  (Akin,  1980),  and  political  science  (Voss, 
Greene,  Post,  A  Penna,  1983;  Voss  A  Tyler,  1981).  Within  these  domains,  researchas  have  continued  to  look  at  the 
steps  and  plans  generated  In  coming  to  a  solution,  but  they  have  also  developed  a  further  Interest  In  the  nature  a 
aganlzation  of  the  knowledge  used  In  the  process  of  problem  solving.  A  ma|a  question  regarding  the  nature  a 
organization  of  knowledge  has  been  how  that  knowledge  and  Its  changes  influence  performance. 

Our  knowledge  of  the  dlffaences  between  novices  and  experts  has  reached  the  point  whae  several  genaal 
statements  can  be  made.  First,  experts  In  any  field  vs  more  able  to  recognize  and  remem ba  typical  conditions  within 
thek  was  of  expertise.  Second,  experts  generally  aganlzs  their  knowledge  by  functional  characteristics  of  problems 
while  novices  as  mae  Ifcsty  to  use  surface  features  to  characterize  problems.**  Thae  have  not  been  a  lot  of  explicit 
conclusions,  however,  about  the  particular  knowledge  skucturee  used  by  experts  and  novices.  Na  has  thae  been 
work  describing  the  particular  changes  In  knowledge  and  procaaslng  behavtor  that  happen  as  a  result  of  a  single 
experience. 

*  This  research  is  supported  In  part  by  the  Army  Res  each  Institute  fa  the  Behavtor  si  and  Social 
Sciences  unda  Contract  No.  MDA-903-86-C-173.  Thanks  to  Ken  Allison  and  Gita  Range ra|an,  who 
provided  representations  fa  the  papa  and  Ideas  about  analyzing  the  protocols. 

"Sss  Chi,  si  si  (1982)  end  Qlseer  (1988)  for  mors  discussion  of  novicstapsrt  dlffsrsncss. 
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Our  primary  goal  la  to  discover  tha  changes  that  Individual  experiences  hava  on  a  problam  aolvar.  In  order  to 
achieve  that  goal,  wa  first  hava  to  And  out  what  knowledge  tha  problem  aolvar  a  tart*  with  bat  ora  aoMng  any  problem 
and  what  knowledge  he  hat  later  to  aolve  a  aim  liar  problam.  While  earlier  work  haa  Indicated  that  "good''  diagnoatlc 
ability  la  a  function  more  of  knowledge  about  the  problem  area  being  dlagnoaed  than  of  general  diagnoatlc  skills 
(Miller,  1075),  wa  And  that  tha  diagnoatlc  skllla  of  novtcee  and  experts  alao  differ,  and  therefore  alao  observe  Initial 
akatsgies  of  problem  aolvera  and  those  ueed  after  a  particular  stparlence. 

In  tha  particular  experiment  to  be  dlecuseed,  wo  had  two  goafs.  Our  Aral  waa  to  And  out  what  knowledge  sub¬ 
jects  at  (Afferent  lev  ala  of  expertise  had  and  to  be  able  to  state  the  problam  solving  skate  glee  used  by  subjects  at 
varying  levels  of  expertise.  This,  wa  fait,  would  give  us  a  good  Idas  of  what  things  experience  teaches.  Baaed  on  our 
previous  work  on  memory  and  problem  solving  (Kolodnar,  1985;  Kolodner  &  Simpson,  1984;  Kolodnar  8  Kdodner, 
1987),  wa  expected  that  differences  would  be  In  both  the  amount  known  and  acceesbillty  (or  organization)  of  known 
knowledge.  Our  second  goal  waa  to  Identify  particular  changaa  over  time  In  each  Individual's  handling  of  sped  Ac  prob¬ 
lems  and  types  of  problems.  The  sequence  of  problems  presented  to  tha  subjects  was  derived  such  that  this  would 
be  poesible. 

The  task  domain  we  have  ehoeen  to  look  at,  diagnosis  of  automotive  problems,  is  interesting  for  several  rea¬ 
son a.  Tha  automobile  engine  Is  a  highly  complex  entity.  It  conelste  of  a  number  of  Interacting  systems  acting  to  pro¬ 
duce  the  car's  motion.  Failures  In  any  component  or  system  of  tha  angina  usually  produce  noticable  symptoms  or 
changaa  In  the  car’s  performance,  but  tha  fall urea  theme  elves  are  seldom  obvious  to  tha  amateur.  In  addition,  a  given 
symptom  can  indicate  numerous  posable  failures  with  In  tha  angina.  The  parson  who  cornea  to  the  shop  with  a  prob¬ 
lem  deecrbee  a  symptom  or  set  of  symptoms  to  the  mechanic,  and  It  Is  the  mechanic's  job  to  further  Investigate  the 
car  to  And  out  which  of  the  many  poasbla  problems  that  could  cause  the  reported  symtom(s)  la  In  fact  raaponsble  for 
It.  Experts  are  much  better  than  novtcee  at  determining  tha  causae  of  automotive  problems.  (As  tha  old  story  goes; 
it's  tan  cants  for  the  screw  and  twenty  dollars  for  knowing  which  one  to  replace.) 

The  domain  la  knowledge-rich,  and  the  depth  of  knowledge  and  ability  to  use  H  are  both  Important  In  making  a 
good  diagnosis.  Schools  teach  about  cars  In  general,  but  alnoa  there  are  eo  many  (Afferent  kinds  of  cars,  each  of 
which  hava  their  own  peculiar  Was,  textbooks  and  schools  cent  teach  everything.  Diagnosing  a  oar  with  a  given  set  of 
symptoms  may  depend  as  much  on  the  age  and  type  of  angina  as  on  the  symptoms  presented.  A  given  failure  can  be 
a  common  cause  of  a  particular  symptom  h  one  angina  and  not  posable  In  another.  Experience  with  (Afferent  types  of 
cars  and  differ  ant  types  of  problems  la  thus  essential  In  gaining  expertise.  Furthermore,  there  are  too  many  typee  of 
cars  (moat  models  change  at  teas!  a  Ittle  every  year)  and  too  much  In  the  aeta  of  manuals  for  Individual  cars  for  a 
mechanic  to  know  everything  about  every  car.  Thus,  It  e  essential  for  the  expert  mechanic  to  draw  hie  own  generali¬ 
zations  about  cars  that  allow  him  to  organize  and  access  knowledge  appropriate  to  any  particular  car  and  problam  he 
la  looking  at. 

In  the  work  reported  hero,  three  student  mechanics  ware  observed  while  diagnosing  car  feJIurae.  Six  problems 
wore  presented  at  weakly  Intervals  and  think -aloud  protocols  wore  collected  while  the  students  worked  and  wore  tan- 
acrbed  and  coded  for  later  analysis.  Each  week  Ihe  Instructor  demonstated  the  oorrect  or  optimum  toubies hooting 
sequence  for  diagnosis  of  the  failure  after  all  subjects  wore  finished.  Thus,  each  student  had  an  opportunity  for  feed¬ 
back  and  an  explanation  of  the  car's  problem  whether  or  not  ha  had  dlagnoaed  It  correctly.  Each  failure  wae  hko- 
ducsd  Into  tha  car  deliberately  and  each  problam  waa  caused  by  only  one  faked  part.  Analysis  of  the  date  focussed  on 
the  knowledge  and  aksteglaa  used  by  students  at  (Afferent  Isveia  of  kalnlng,  how  that  knowledge  was  organized,  and 
how  their  knowledge  and  strategies  changed  with  experience. 

We  axpected  that  the  mors  experienced  student  would  aolve  more  problems  and  would  give  evidence  of  having 
a  more  organized  knowledge  baas  than  the  Isas  experienced  students,  in  addition,  wa  expected  that  Indhrlduale  would 
show  evidence  over  the  aarlao  of  problem  of  acqukbg  new  dtagnostic  Mia  and  new  knowledge  and  connections 
within  their  knowledge. 
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1.  Method 


l.t.  Sub  Joe  to 

Throe  students  at  a  poet  secondary  technical  achool  volunteered  to  participate  In  the  project.  The  technical 
program  la  a  two-year,  eight-quarter  program.  During  much  of  the  second  year,  the  students  work  In  a  shop  setting 
within  the  achool.  Cars  belonging  to  school  personnel  and  friends  of  the  students  and  hsfructors  are  dtagnoeed  and 
repaired  by  students.  In  addition,  the  school  owns  several  cars  that  can  be  used  in  teaching  students  to  teach  about 
specific  problems. 

Each  of  the  three  student  volunteers  was  at  a  different  point  In  the  program.  The  novice  student  was  In  hie  first 
quarter  of  the  program  and  had  no  prior  training  or  experience.  The  intermediate  student  was  at  the  beginning  of  hie 
second  year  In  the  program.  The  advanced  student  was  near  the  end  of  the  second  year  and  held  a  part-time  Job  as 
a  mechanic  outside  of  school.  Each  student  worked  on  at  least  tour  of  six  problems. 

2.2.  Procedure 

Subjects  ware  observed  once  a  week  while  diagnosing  an  actual  problem  In  a  car.  The  problems  uaed  were 
selected  by  an  hsfructor  tn  the  program  In  consultation  with  the  experimenter.  The  problems  and  the  Information 
given  as  the  customer's  complaint  are  deecrbed  In  Table  1.  Each  fault  was  Introduced  Into  a  car  by  the  tnefructor  or 
by  a  student  not  tn  the  study  under  the  direction  of  the  Instructor.  The  cars  used  were  at  owned  by  the  school  with 
one  exception:  a  new  car  brought  In  by  a  school  official  that  had  symptoms  we  had  been  presenting  to  the  students  In 
previous  weeks.  In  every  case,  a  single  complaint  was  given  and  a  single  fault  could  be  fcacad  to  account  for  the 
complaint.  Students  were  told  to  back  down  the  fault,  but  not  to  fbc  It  unless  repair  was  necessary  to  confirm  the  diag¬ 
nosis. 

In  each  session,  the  student  was  led  to  the  car  and,  with  the  experimenter  posing  as  a  customer,  told  that  the 
oar  was  exhibiting  a  particular  symptom.  The  student  was  than  allowed  to  perform  any  taels  desired  on  tie  car  and 
Its  engine,  with  the  exception  of  a  driving  road  test,  prohbltsd  primarily  by  the  symptoms  presented  by  the  car.  The 
student  was  insfructed  to  think  aloud  as  he  worked  to  find  the  (sled  component  in  the  oar.  Hia  comments  ware  tape 
recorded  by  the  experimenter,  who  also  served  as  an  assistant  to  the  student  when  necessary. 


Table  1 

Faults  and  their  complaints  as  presented  to  subjects 

Problem 

Complaint  (Symptom) 

Fault 

1 

cranks  but  w*  not  start 

sediment  or  other  blockage  In  gas  Ine 

2 

cranks  slowly  when  starting 

bad  ceil  b  battery-will  not  hold  charge 

3 

cranks  but  wW  not  start 

bad  connection  behind  fuse  panel  and 
fuel  pump  fuse 

4 

cranks  but  wff  not  start 

loose  ground  wires  from  Elecfronlc 
Conboi  Module  (oomputer) 

5 

cranks  but  wM  not  start 

open  tach  ctcult 

6 

detonation  on  acceleration 

poorly  adjusted  timing 

2.2.  Ceding 

After  al  protocola  ware  franecrlbed,  each  statement  was  coded  Into  one  of  ate  categories,  shown  In  Table  2 
with  example*.  Statements  coded  as  hypotho»o§  war*  those  In  which  a  specific  system  or  component  was  ftst 
named  as  a  posable  source  of  the  (allure  or  In  which  the  system  or  component  was  aocepted  or  rejected  as  the 


.1 
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Table  2 

Coding  Categories  for  Protocols 

Category 

Additional  Specifications 

Examples 

Hypotheses 

Number  and  Status 

Could  be  starved  for  gas  (N-Pl) 

It  could  be,  could  be  the  starter  (N- 
P2) 

Rules 

Topic (Fai lure,  normal  functioning,  or 
troubleshooting) 

Fuel  Pump  should  come  on  for  3 
seconds  (I-P4) 

First  of  all,  1  have  to  locate  the 
connector  to  the  back  of  the  fuel 
pump  (A-P3) 

Information  Gathering 

Source  of  ^formation  obtained 

Before  t  look  In  the  book,  I'm  going  to 
check  the  fuse  (A-P3) 

Observation 

Topic  (hypothesis(number)  or 

complaint) 

What  we  don't  have  is  fuel  to  the 
throttle  body  (A-P3) 

1  don't  believe  1  hear  It  running  (A-P3) 

Restatements 

Topic  (complaint  or  summary  of 
observations) 

to  rephrase  that-the  throttle  body  Is 
not  injecting  fuel  (A-P3) 

source  of  the  failure.  Hypotheses  ware  numbered  tn  order  of  appearance  and.  aach  time  ona  waa  mentioned,  Ha 
atatua  was  noted.  Its  status  could  be  open,  accepted,  confirmed,  or  refected.  Rules  wars  statements  gMng  known, 
oonatant  Information  about  an  angina  or  about  the  process  of  diagnosis.  Statements  coded  aa  Information  gathering 
were  generally  descriptions  of  the  actions  being  taken  by  the  subject  at  the  time.  Such  actions  could  elicit  or  obtain 
Information  from  the  customer,  from  a  book,  or  via  a  procedure  or  test  applied  to  the  angina.  Observation*  were 
statements  giving  the  Information  obtained  from  the  action  taken.  Restatement*  were  repetitions  of  previously  stated 
or  collected  Information  rather  than  now  Information.  Each  statement  falling  Into  one  of  the  last  three  categories  was 
identified  with  a  specific  hypothesis  by  Its  number  If  poesfcle.  All  other  statements  were  uncodab le  and  were  marked 
as  such. 


9.  Results 


As  expected,  the  ability  of  the  students  to  correctly  diagnose  the  problems  changed  aubstantlally  between  the 
novice  level  and  the  htermediate  and  advanced  levels.  The  diagnoses  given  by  each  subject  and  the  number  of 
hypotheses  considered  are  shown  in  Table  3.  The  novice  correctly  diagnosed  only  one  of  four  probieme  attempted, 
while  the  Intermediate  student  correctly  diagnosed  three  of  six  and  the  advanced  student  three  of  four.  In  addition,  the 
number  of  hypotheses  considered  Increased  with  expertise.  The  novice  generated  a  moan  of  3.0  hypotheses  per 
problem  and  the  Intermediate  and  advanced  students  generated  6.B  and  5.0  hypotheses  per  problem  respectively. 


3.1.  Knowledge  Structure*  and  Knowledge  Organization 

In  general,  the  diagnostic  behavior  we  aaw  was  elmllar  to  that  reported  by  other  reeeerchere  (Hunt,  1981; 
Rasmueeen,  1978;  1979;  Rasmussen  3  Jansen,  1974).  8 tu dents  generated  one  or  more  poeeble  hypotheses  tor  the 
(allure  Immediately  after  observing  the  symptom (s).  These  hypotheses  were  then  tested  In  a  fairly  systematic  (albeit 
sometimes  Idiosyncratic)  way  either  by  observation  of  the  Inputs  to  and  outputs  from  specific  components  and  sys¬ 
tems  or  by  performance  of  specific  diagnostic  tests.  In  successful  cases,  a  single  diagnosis  ultimately  was  given, 
accompanied  by  an  explanation  of  how  or  why  that  failure  would  generate  the  observed  eymptom(s). 

We  Interpret  this  process  as  being  Indicative  of  an  Interaction  between  two  types  of  knowledge  structures.  The 
first,  a  causal  modal  of  the  car's  engine,  contains  knowledge  about  Individual  components  and  their  Inputs,  outputs, 
and  normal  behavior;  relates  components  within  a  system  to  one  another;  and  deecrbes  the  relationships  and  connec¬ 
tions  between  systems.  It  Is  used  to  evaluate  hypotheses  In  Rght  of  the  evidence  obtained  from  the  failed  engine  and 
to  lead  the  mechanic  through  the  engine  to  the  source  of  the  problem  In  a  systematic  way.  The  causal  modal  is  gen¬ 
erally  quite  large,  and  the  second  type  of  knowledge  structure,  symptom-fault  sets.  Is  used  to  Index  Into  the  causal 
model  at  appropriate  pieces.  8ymptom-f*ult  sets  represent  the  relationships  between  particular  symptom*  or  sets  of 
symptoms  and  falluras.  For  example,  given  the  symptom  "the  car  cranks  but  will  not  start",  the  symptom-fault  sets  will 
Identify  three  systems  as  possible  locations  for  tha  failure:  the  fuel  system,  the  air  Intake  system,  and  the  Ignition  sys¬ 
tem.  Within  each  of  theee  systems,  additional  symptom-fault  aets  will  Identify  Individual  components  that  may  cause 
the  symptom(s).  For  the  fuel  system,  theee  would  be  a  failed  fuel  pump,  an  empty  gas  tank,  or  a  blocked  fuel  line. 
For  the  Ignition  system,  these  would  be  a  bad  dislrbutor,  bad  apark  plug  wfres,  or  bad  apark  plugs.  These 
symptom-fault  sets  are  ueed  to  derive  Initial  hypotheses,  directing  the  mechanic  to  look  at  only  appropriate  places  in 
the  causal  model. 

If,  In  fact,  mechanics  are  using  these  two  types  of  knowledge  sfructures  during  troubleshooting,  then  we  can 
predict  several  changes  we  should  expect  to  see  In  these  sfructures  as  a  result  of  experience,  and  from  those,  we  can 
predict  the  processing  differences  that  would  result  from  these  changes.  First,  we  predict  that  through  experience,  a 
mechanic's  aet  of  symptom-fault  seta  Increases  and  that  tie  sets  he  steady  knows  become  more  accurate.  As  a 
result  of  these  changes,  the  mechanic  should  have  better  ways  to  Index  into  the  causal  model,  loading  to  more  effi¬ 
cient  searches  for  the  correct  failure.  Second,  the  causal  modal  should  become  more  filled  out  with  experience,  both 
through  addition  of  components  and/or  systems  that  wars  previously  unknown  and  through  addition  of  relation  ship* 
and  dependencies  between  the  known  components.  The  caueal  model,  See  symptom-fault  aets,  should  also  become 
more  accurate.  As  a  result  of  havhg  a  better  causal  model,  a  mechanic  should  be  better  able  to  systematically  reason 
about  the  way  the  car  works,  allowing  him  to  find  engine  failures  more  systematical  and  in  more  cases. 

We  dkf,  fri  fact,  sse  dear  deference*  between  students  at  different  levels  of  experience  reflecting  exactly  theee 
changes  fri  thefr  knowledge  sfructures.  First,  we  saw  evidence  that  both  the  organization  and  number  of  symptom- 
fault  sets  Increased  with  experience.  The  advanced  student  seemed  to  know  mors  symptom-fault  set*  than  the 
novice,  as  evidenced  by  the  larger  number  of  hypotheses  he  was  able  to  generate  fa  each  problem.  In  addition,  the 
advanced  student  seemed  to  organize  his  symptom-fault  sets  differently  than  the  novice,  evidenced  by  the  more  sys¬ 
tematic  procedure  he  ueed  for  generating  and  testing  hypotheses.  The  advanced  student's  procedure  was  to  zero  In 
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oo  one  of  the  engine's  subsystems  end  then  to  consider  which  component  of  that  system  was  faulty,  while  the  novice 
did  not  differentiate  between  systems  and  components  of  systems  in  diagnosis.  While  for  the  novice,  all  faults  are 
equal  and  an  hypothesis  at  the  component  level  was  as  Ikely  to  be  selected  as  the  first  to  Investigate  as  an  hypothesis 
at  the  system  level,  the  more  advanced  trouble  shoo  tar  seemed  to  organize  his  symptom-fault  sets  No  two  categories, 
each  used  for  different  purposes.  One  set  pointed  to  faulty  subsystems  within  the  car  (e.g.,  fuel  system,  electrical 
systems)  and  was  used  early  m  diagnosis  to  zaro  in  on  the  faulty  subsystem,  while  the  second  set  pointed  to  faulty 
components  of  these  systems  (e.g.,  the  fuel  pump,  the  battery)  and  was  used  to  diagnose  the  problem  within  that 
system.  Such  a  change  requires  that  the  mechanic  also  reorganize  hie  knowledge  about  the  car's  engine  in  a  more 
hierarchical  way  that  differentiates  between  systems  and  components  of  systems.  Figure  1  shows  a  portion  of  the 
novice  and  advanced  student  s  organizations  of  the  causal  model  of  the  engine. 
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Figure  1 

Novice  and  Advanced  Student  Representations  of  the  Car's  Engine 


We  also  saw  evidence  that  content  of  the  causal  model  changed  wttti  experience.  The  cauaal  model  of  the 
more  advanced  students  contained  not  only  more  knowledge  about  Individual  components,  but  also  more  knowledge 
about  the  interconnected  nature  of  the  engine's  systems.  The  behavior  of  the  students  during  troubleshooting  Illus¬ 
trates  these  findings.  Consider,  for  example,  the  behavior  of  the  advanced  student  in  Problem  4.  His  reasoning  went 
as  follows:* 

The  ft st  thing  you  want  to  do,  which  la  the  easiest  thing  to  do,  Is  look  and  see  If  we  have  any  fuel, 
because  you  gotta  have  fuel,  air,  and  heat...  Dont  have  fuel. .The  frst  thing  I  want  to  do  is  check 
the  fuse... they're  OK...  hook  this  jumper  lead  to  the  bypass  to  the  fuel  pump.. .the  fuel  pump  la 
running...  check  and  sea  our  connection  up  here  to  the  energizer. ..going  from  the  ECM  up  to  the 
fcjector  is  OK. ..fry  to  energize  this  solenoid  by  hand.. .check  to  see  If  we  got  any  gas. .an  the  Hnee 
ate  aHght...got  gas  to  the  throttle  body...  my  diagnosis  is  the  solenoid  is  bad  because  everything 
else  checks  out. 

The  hypotheses  generated  by  this  student  are  in  an  order  that  reflects  the  multi-level  and  highly  Integrated 
*  For  a  lull  protocol  of  the  session,  writs  to  the  first  author. 


organization  of  both  hia  cauaal  modal  and  hie  symptom -fault  eats.  He  fret  determined  which  ot  three  possble  sys¬ 
tems  of  the  engine  was  affected  and  then  Investigated  tte  components  and  others  that  could  impinge  on  the  behavior 
of  the  system  under  focus.  In  fact,  hie  primary  focus  was  on  the  electronic  (or  computer  controlled)  Influences  on  the 
behavior  of  the  fuel  pump  and  fuel  Injectors.  This  reasoning  showed  an  awareness  (reflected  In  the  student's  causal 
model)  of  the  Interdependencies  between  subsystems.  Hie  reasoning  shows  that  he  knows  that  systems  (such  as  the 
fuel  and  etocfronlc  systems)  may  Intersect  at  several  points  and  that  an  apparently  or  poesbty  failed  component  In  on6 
system  may  reflect  an  action,  or  lack  of  action  In  another  system. 

In  contest,  the  novice  generated  relatively  few  hypotheses  for  any  given  problem.  His  protocols  indicate  that 
this  Is  because  he  has  little  knowledge  about  the  relationships  between  given  symptoms  and  their  causes  and  also 
because  hia  causal  model  la  inadequate.  In  solving  the  same  problem  the  advanced  student  was  working  on  above, 
the  novice  reasoned: 

This  problem  could  be  h  the  fuel  system,  Ignition  system. ..ere  know  It's  not  In  the  starting  system 
because  the  car  will  crank  over. ..One  small  cfrop  of  fuel. ..In  that  bowl. ..so  It’s  In  the  fuel 
aystem...the  fuel  pump’s...  supposed  to  turn  for  10  to  IS  seconds...!  cant  hear  H...lt  might  just  be 
a  bad  fuel  pump. 

Wa  can  see  wttte  evidence  of  an  in  tape  tad  hierarchy  of  terete  in  hia  organization  of  symptom-fault  seta.  While  his 
hypotheses  were  some  times  at  tha  system  level  (l.a.,  fuel  system)  and  sometimes  at  the  component  level  (l.e.,  fuel 
pump  is  bad),  h  only  one  problem  (this  one)  did  ha  dearly  consider  fkat  a  system  and  than  a  component  within  that 
system.  Mora  commonly,  ha  generated  hypotheses  at  both  lev  ala  and  than  Investigated  only  specific  components. 
Furthermore,  he  showed  a  similar  lack  of  integration  In  hia  causal  model.  Specifically,  he  never  cone! derad  the  possi¬ 
bility  that  one  system  could  affect  the  behavior  of  another.  His  knowledge  appeared  to  stop  at  the  individual 
component's  behavior  and  did  not  Include  the  possfclHfy  that  the  actions  of  another  system  (the  electronic  system) 
could  be  affecting  the  behavior  of  the  component  he  was  ooneidarlng  (the  fuel  pump). 

While  the  novice  knew  about  many  of  the  components  of  the  car 'a  angina  and  about  what  their  connections 
were  within  a  single  system,  he  did  not  know  how  the  systems  and  tha  components  In  different  systems  ware  Interre¬ 
lated.  The  advanced  student,  on  the  other  hand,  knew  both  the  connections  between  components  and  the  connec¬ 
tions  between  systems.  Thus  the  advanced  student  had  a  mors  intagrated  and  oomplete  understanding  of  the  car  e 
engine,  while  the  novice's  understanding  seemed  to  be  highly  disjoint.  Figure  2  shows  our  Interpretation  of  what  the 
novice  and  advanced  students  knew  about  the  fuel  pump,  for  example. 


PUMP  Source:  a  container 

Substance:  a  substance  In  the  container 
Conduit:  a  pipe 
Destination:  a  container 
Energy -Sour  oe:  an  energy  device 
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Substance:  gasoline 
Conduit:  hose 
Destination:  carburetor 
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Figure  2 

Novice  and  Advanced  Student  Representations  of  a  Fuel  Pump 
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Note  that  the  general  Information  about  pump#  ie  available  to  both  the  novice  and  the  advanced  etudent.  How¬ 
ever,  the  Information  that  the  fuel  pump  requires  an  energy  source  which  ie  the  eiectlcal  eyatem  of  the  car  ie  not  part 
of  the  novice  #  representation  of  the  fuel  pump.  If  asked  "What  makes  the  fuel  pump  run?",  the  novice  I#  able  to  con¬ 
struct  the  appropriate  answer  by  using  the  more  general  information  about  pumps,  but  he  does  not  use  this 
knowledge  during  problem  solving.  The  same  pattern  Ie  probably  Sue  of  knowledge  about  aysteme  and  components. 
The  novice  can  undoubtedly  tell  an  Inquirer  what  system  of  the  engine  a  particular  component  resides  h,  but  he  does 
not  maintain  this  information  where  It  Is  readily  usable  during  problem  solving. 

We  also  saw  within -subject  changes  In  these  knowledge  structures  over  the  course  of  the  experiment.  These 
changes  were  most  evident  In  the  intermediate  student.  Two  examples  will  serve  to  demonstrate  changes  across 
problems.  In  working  on  problem  three,  the  Intermediate  student  made  a  long  and  protracted  search  for  the  fuel  pump 
relay  using  both  written  reference  materials  and  extended  visual  examination  of  the  engine.  While  working  on  problem 
four,  he  was  able  to  Immediately  locate  and  check  the  same  part.  This  component,  and  Its  physical  relationship  to 
others,  had  been  Incorporated  Into  the  causal  model  during  or  following  problem  three.  Similarly,  the  symptom-fault 
sets  changed  as  new  Information  was  acqursd.  For  example,  the  first  hypothesis  the  Intermediate  student  checked  at 
the  component  level  for  problem  four  was  the  fuel  pump  fuse,  which  was  the  correct  diagnosis  for  problem  three.  He 
made  the  point  as  he  worked  that  he  was  checking  this  possibility  out  first  because  of  the  previous  case.  ("I'm  gonna 
check  the  fuel  pump  fuse  first  [this  time].") 

3.2.  Diagnostic  Strategies 

In  addition  to  the  changes  experience  makes  In  knowledge  structures  and  organization,  we  also  saw  differences 
In  diagnostic  style.  Diagnostic  strategies  seemed  to  be  used  differently  by  subjects  at  different  levels  of  expertise  and 
evaluation  criteria  changed  significantly  with  experience.  Some  of  these  changes  are  due  to  the  development  of  better 
strategies  for  testing  and  confirming  hypotheses  with  experience  while  others  appear  to  result  from  the  differences  In 
the  knowledge  available  for  diagnosis  as  a  mechanic  gets  more  experienced. 

The  change  In  how  the  mechanics  tested  and  confirmed  hypotheses  was  strking.  As  the  example  above 
showed,  the  novice  student  was  willing  to  accept  an  hypothesis  when  preliminary  evidence  could  be  Interpreted  as 
congruent  with  that  hypothesis  and  not  pursuing  the  task  any  further  (1-8.  "cant  hear  the  fuel  pump").  In  contrast,  the 
advanced  student  sought,  for  each  hypothesis,  specifically  confirming  or  discon  firming  evidence  that  was  part  of  a 
causal  explanation.  While  he  was  willing  to  select  an  hypothesis  to  pursue  on  the  basis  of  preliminary  evidence,  he 
would  not  accept  or  reject  It  without  causally  based  information  (l.e.  "the  fuel  pump's  not  running,  now  we  have  lo  find 
out  why"). 

The  changes  in  diagnostic  strategies  that  resulted  from  changes  in  the  knowledge  sfructures  were  more 
apparent  In  the  efficiency  of  diagnosis.  As  the  causal  model  gets  filled  out,  It  should  allow  the  mechanic  to  pursue  a 
longer  systematic  search  through  the  engine  and  also  allow  him  to  evaluate  information  in  more  detail  and  with  more 
concern  for  the  real  affects  of  the  behavior  observed.  At  the  same  time,  as  the  number  and  complexity  of  symptom- 
fault  sets  increases,  long  searches  should  become  less  necessary,  because  the  mechanic  is  able  to  index  into  his 
model  In  more,  and  more  effective,  locations. 

These  two  types  of  changes  In  the  mechanic's  diagnostic  strategies  work  together  to  produce  the  results  we 
saw.  As  the  mechanic  gains  experience  wtth  making  cored  and  In  cor  eel  diagnoses,  he  gains  a  sense  of  what  kind 
and  how  much  Information  Ie  "enough"  to  be  sure  of  hie  opinions.  In  addition,  as  his  causal  model  and  symptom-fault 
sets  become  more  complete  and  accurate,  he  is  mas  able  to  select  hypotheses  tor  Investigation  appropriately  and  to 
continue  Invgeetigating  a  problem  to  the  point  that  only  one  hypothesis  remains  as  a  possible  diagnosis.  Conse¬ 
quently,  th«  conditions  under  which  he  will  accept  an  hypothesis  as  a  final  diagnosis  will  become  more  accurate  and 
the  path  by  which  he  reaches  his  die  gnosis  wfli  become  mas  efficient. 

This  result  is  dearly  evident  In  protocols  of  the  novice  and  advanced  students.  When  the  novice's  working 
hypothesis  was  a  that  a  particular  component  was  faulty,  he  either  accepted  It  or  rejected  It  as  the  cause  of  the  symp¬ 
tom.  He  never  Investigated  other  effects  on  or  Inputs  to  that  component.  Fa  example,  tin  problem  2,  the  failure  was  a 
dead  battery  cell  which  caused  the  car  to  crank  very  slowly.  The  novice  based  his  diagnosis  on  the  following  informa¬ 
tion: 
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Fkst  of  all,  wall  have  to  check  this  battery. ..It  could  be  the  etartar...  H  could  be  the  alternator. ..It 
could  be  a  voltage  loss. ..could  be  a  dead  cell  In  the  battery. ..we've  only  got  10  volts  in  the  battery- 
-each  battery  cell  Is  2  volts  and  there's  6  oslls  m  the  battery,  so  dead  battery  cell. 

Here  we  see  the  novice  generating  both  system  and  component  level  hypotheses  but,  because  his  knowledge  is  not 
hierarchically  organized,  not  pursuing  them  to  that  order.  Rather,  he  locks  first  at  the  battery  charge.  Because  It  Is 
low,  he  accepts  the  hypothesis  of  a  dead  oell.  His  diagnostic  sfrategy  does  not  require  that  he  consider  any 
hypotheses  relating  to  why  the  battery  might  be  low,  such  as  a  malfunction  in  another  system. 

In  oontrsst,  the  advanced  student  generally  collected  mors  Information  before  giving  a  diagnosis.  If  possible,  he 
confirmed  hie  diagnosis  by  visually  finding  the  condition  that  crested  the  symptom  (La.,  the  disabled  fuse  panel  con¬ 
nection  In  Problem  3).  When  that  was  not  possible,  he  Justified  his  diagnosis  within  his  causal  model.  For  example,  In 
Problem  2,  the  (allure  could  not  be  confirmed  by  visual  evidence.  Instead,  the  advanced  student  reaches  his  diag¬ 
nosis  with  the  following  Information: 

...check  the  starter  daw. ..It's  puMng  enough  down  to  get  the  starter  to  go  alright.. .We  put  the  bat¬ 
tery  under  load,  you  can  see  the  amps  rising  and  It’s  charging  the  battery. ..So  the  alternator's 
working  OK. ..what  I  believe  we  have  Is  the  cell  Is  dead  In  the  battery.. .Try  toe  test  on  the  VAT.. .As 
you  see  on  toe  Indicator  Is  also  showing  that  It  needs  charging  for  the  battery  Is  bad. ..So  what  we 
have  here  is  a  battery  with  a  couple  of  oeAs  dead,  and  It's  a  sealed  battery  and  you  cannot  check 
toe  specific  gravity  with  a  hydrometer  to  check  and  see  which  one's  dead. 

He  reached  and  Justified  his  diagnosis  by  eliminating  all  other  possibilities  from  his  symptom-fault  sets  and  toe  causal 
model.  In  other  words,  he  tested  and  verified  normal  functioning  of  both  the  starting  system  (  It's  puling  enough 
down  to  get  toe  starter  to  go  aUght")  and  toe  charging  system  ("So  the  alternator's  working  OK").  These  arc  the  only 
two  systems,  other  than  accessories  such  as  headlights  and  radio,  that  affect  the  level  of  charge  In  the  battery.  Con¬ 
sequently,  according  to  the  student's  causal  model,  If  toe  battery's  charge  Is  low  and  the  starting  end  charging  sys¬ 
tems  are  functioning  correctly,  the  only  remaining  component  In  which  the  (allure  can  be  located  Is  toe  battery  Itself.  In 
some  types  of  batteries,  this  conclusion  can  be  tasted  directly,  but  In  toe  car  used  In  this  problem,  toe  battery  Is 
sealed.  Therefore,  the  mechanic  must  stop  with  his  explanation  rather  than  attempt  to  verify  the  diagnosis  any  further. 
In  comparison  to  toe  novice,  he  selected  his  hypotheses  more  efficiently,  drat  eliminating  competing  systems  from  con¬ 
sideration.  In  addition,  he  based  his  aoceptsnce  of  the  diagnosis  on  a  full  causal  explanation  rather  than  on  superficial 
evidence. 

4.  Conclusions 

The  results  are  as  predicted  by  our  Interpretation  of  the  dtepiostlc  behavior  as  an  Interaction  between  several 
knowledge  structures.  Both  the  causal  model  and  toe  symptom-fault  sets  change  with  experience,  and  we  have  seen 
some  examples  of  exactly  what  changes  occur.  In  toe  causal  modal,  the  most  notable  change  Is  toe  Increasing  com¬ 
plexity  of  the  model,  reflected  In  the  growing  awareness  of  the  Interconnectedness  of  systems  within  toe  engine.  The 
novice  is  dearly  unaware  of  the  poeebllity  that  electronic  (allures  can  affect  things  Ike  fuel  delivery,  since  he  knows  lit¬ 
tle  about  toe  dependencies  between  toe  fuel  system  and  the  si  act  leal  system,  while  the  mas  advanced  mechanic  not 
only  knows  that  such  relationships  exist,  he  considers  them  a  highly  common  source  of  failures.  Slmtarly,  toe 
number,  organization,  and  accuracy  of  toe  symptom-fault  sets  changes  wfto  Increasing  experience.  Ultimately,  they 
we  able  to  represent  a  complex,  hierarchical  system  of  relationships.  The  data  suggest  that  components  a re  organ¬ 
ized  hierarchically  under  their  respective  systems  and  are  never  dkectfy  considered  unless  that  system  Is  determined 
to  house  the  (allure,  or  at  least  to  be  the  source  of  information  crucial  to  locating  toe  (allure. 

Building  partly  on  these  changes  In  the  knowledge  sfructuras,  and  partly  on  ^dependent  effects  of  expwienoo 
on  decision  processes,  the  mechanic's  procedurss  and  guidelines  for  aooepttog  hypotheses  as  diagnoses  also  change. 
The  processes  or  procedures  used  become  Increasingly  focussed  on  Information  that  wM  alow  a  causal  Interpretation 
of  toe  behavior  observed.  At  toe  same  time,  the  developing  knowledge  sfructures  ak>w  the  mechanic  to  search  for 
and  aqulre  more,  and  more  accurate,  Information  from  his  symptom-fault  sets  and  his  causal  model.  The  h terse tion 
of  these  changes  In  both  knowledge  and  process  lead  to  the  more  accurate  and  efficient  problem  solving  seen  In 
experts. 


Thu*,  w*  m  that  experience  I*  providing  the  mechanic  with  three  things.  Hie  overall  level  of  knowledge  Is 

Increasing;  the  organization  and  Integration  of  his  knowledge  structures,  both  the  symptom-fault  sets  and  the  causal 

model,  are  Increasing;  and  hie  processes  and  criteria  for  reaching  diagnoses  are  becoming  more  accurate,  more  effi¬ 
cient,  and  more  focussed  on  causal  Information. 
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